it had been a great flight with a great
entry in landing and the team was
preparing to handover the vehicle to the
KSC ground folks one of last orders of
business in this process is to
transition from internal vehicle cooling
to ground cooling now in orbit cooling
is accomplished through a combination of
the radiators and flash evaporator
system as shuttle equipment heats up the
freon loops and water loops carry this
heat to the flash evaporator and
radiators to be dispersed into space for
entry the radiators are cold soaked and
then bypassed so that that cooling can
be used on the ground then post landing
the shuttle uses its cold soaked
radiators for a few minutes and then as
they heat up the team activates the
ammonia system which flashes ammonia
onto the freon loops to provide cooling
in order to have a safe transition to
ground cooling the ammonia system has to
be deactivated and confirmed to have no
leaks and then a go is given so the
ground team can activate ground cooling
at least that's the way it's supposed to
go now there are some critical elements
to consider regarding cooling obviously
you have to keep cooling going so
nothing overheats but at the same time
you can't get too cold or you could
freeze the interchange the interchange
er is the heat exchanger where the water
loops dump the cabin heat onto the fan
loops all the shuttle cooling loops meet
there and since we're dealing with water
we can't let it get below 32 degrees
Fahrenheit or the water loops could
freeze and as you know when water
freezes it expands and cracked
you've just ruptured a piece of
equipment that could cost over a million
dollars to replace this is a very
efficient heat exchanger so this would
only take a matter of seconds on sts-27
the e-comm team was faced with just this
scenario
I was going to see the it's post-landing
and everything's going normally you're
on the ground you've made it right
what could go wrong here the team is
even starting the post-landing tradition
of hanging the plaque in which one of
the flight controllers is honored with
special recognition however not
everything is done handover to ground
cooling hasn't been completed and the
ecomp team is watching ammonia
performance as suddenly the friendly
temperatures start diving so I want you
to tell me first of all we got Queen
Caroline and Chuck dangle for sts-27 I
want you to tell me your positions for
this flight and what the
responsibilities those positions were so
start with the leader chuck of the team
okay so it's been a while but I think it
was a son entry in orbit one II come for
the flight so we were responsible for
the life support thermal control and
also at that time the electric power
systems on onboard okay I was the acid
entry thermal specifically responsible
for the active cooling and passive
cooling systems on the shuttle including
the ammonia boiler freon and warm loops
okay so what were you looking at there
that cued you into the fact hey I got a
problem going on here was that that bill
that went off the alarm that went off
earlier probably actually what happened
was the first thing I saw was we have an
indication on the ammonia system the
primary controller if something goes
along with the logic or goes under temp
it automatically switches to the
secondary controller trying to go to a
different controller to control the
system yet that was the first insight as
I saw is this indication that we just
switched from the primary controller to
the secondary controller and generally
means under tempo that is the only
indication at the time was seeing a
switch and having said that though it's
upstream and we have a power T boosters
so the temperatures hadn't arrived at
yeah how many seconds is that from
ammonia I forgot yeah it's about 45
seconds or so yeah so you guys knew
something something switched over but
you probably didn't have an idea at that
point it was an under tamp right okay
okay and I think he may have just called
hey I think he called it under Tim
switch over maybe okay
now the team has confirmed there's an
under tint problem the cold slug has now
shown up on the downstream temperature
dosers and the extra cold freon is
racing towards the inner changer which
could crack a face with these
temperatures how long does the team have
to respond what is the correct course of
action to take okay so at this point you
know that you've got the about too low
and what are you thinking one of my
thing at the time there was a hairy edge
where you know as the I knew that it may
be the initiated ground : there's some
spec leakage to the ammonia boiler that
it leaks ammonia so you're getting some
cooling from it whether it's on or when
it's on no matter what if it's not
required on it so it's providing extra
cooling to loop
initial thought was okay we're down just
broke 32 degrees maybe they're cool and
they're cooling cards just to go low am
i right now
ready to take this drastic action yet or
is this maybe we can take the ammonia
boiler off and that would be sufficient
to bring it above freezing again what's
my initial thoughts it was not it was
pretty getting pretty excited you can
hear was the fact that you know we were
actually below the freezing point of the
water and we needed to do something
quick
so that was an initial reaction let's
get it off and see if that won't bring
it to the trip back up above freezing so
what's what is the issue here with
getting too cold what's the what's the
problem with doing that the interchange
is a very efficient heat exchanger and
it's got freon on one side and water on
the other and if you put in freon below
32 degrees in pretty short order you're
going to end up freezing water turn the
water into ice it expands and volume
change in heat exchanger can break
that'd be pretty expensive to try to
replace that and one of the big things
was that we have done that before and
sts-3 it was a better place the big
thing is I don't think they have any
spares and I think the big thing is it
just bits of not only the cost which
they estimated 100 million dollars but
getting a new one in installing it from
the downtown of the vehicle was a big
impact impact of schedule would have
been pretty enormous right I mean
basically can't power how vehicle what
the real move or loosen on actors so
knowing that why didn't she just go take
the drastic it what is drastic about
turning the friendly soft ones just say
let's go ahead and turn them off what's
what's the issue there you know so we
got on one side we don't want to freeze
the interchange er why don't we want to
just turn the family piece off what's
the issue there well I kind of had my
way of saying who gets below 30 degrees
that's what my land my saying we're not
playing anymore and he's you're gonna
hearing a little bit it happened quickly
from drop down and it was very officer
obvious that once it got to obstacle low
of the deucer which was twenty four
point nine you'll hear that several
times repeated very obvious what we need
to do but it was to me was a drastic I
mean the fact that we potentially shut
it down we may not bring it back up
that's okay so basically you're okay
sounds like nothing you made the call
for the frame loops nothing happened
right initially and and I'm still like
they're you know they're actually the
trawl team was basically and like I said
the post mission mode where they're
doing the post black hanging and stuff
and I think even GC got to the point or
actually we temporarily they shut down
calm between the vehicles what I heard
later for the infirm their first initial
call to the vehicle to turn off both
trailers and I don't think that we left
the house they're probably thinking the
rule the one point one hey the crews
gone why should I keep calm up I mean
we're already doing the plaque hanging
right let's just let's just shut this
thing down so okay I don't know what I
don't know what all the factors were in
that listening to that recording which I
don't think I've ever listened to
just being kind of stepping back as an
observer to myself if I would have had
the wherewithal I think it would have
helped to be more exact in the call
especially knowing that the flight crew
was not in there
who probably could have responded very
well to be activate both freon loops
were the support person in there instead
a more precise call like on panel l1
freon pump one off loop - I think that
in hindsight would have helped get that
done much faster if I was a ASP in the
cockpit trying to interpret that call I
probably have trouble myself right here
ms Wanda had seemed in a year where
that's good that's a very good point
that's a good point okay so what was
going on there I made the call for the
Fran loops and now Quinn's calling again
let's get one off yeah so I can tell you
from my perspective I was trying to find
the procedure for a VAT Bell tell about
temple oh I think in the entry checklist
right okay so I was thumbing through the
book to fly to to try and find that
procedure to save myself so I after
making the call to turn both freon loops
off I think I was under the assumption
that that would happen quickly I'm going
to find the procedure to look for
subsequent steps to guide us and so
Quinn called and I'm asking questions I
think out of that procedure and Quinn
says hey I just please turn off it hit
me at that point
oh they didn't respond to the call right
and you Quinn can't see each other here
right ones in one room ones in the other
room so and sometimes there's over the
airwave conversation that goes on and so
you didn't know why they weren't turning
the friendly
right and I'd take from the is a mid
perspective one thing that really taught
me it was okay I've already made the
call
I'm watching Chuck Chuck is watching
this data to don't need a bother he sees
what's happening I'm kind of in the
background dancing panicking but I am
NOT I learned after the fact that when
he started talking about what I want to
do next it was a fear through fear
through mean that he doesn't realize
they're still not off yet
and then I'm questioning why didn't they
go off why didn't they they actually do
it so but the point in listening for
future guys is you know don't make an
assumption you're free clickers look in
the data this is all växjö it's
happening much quicker than it really is
but we had I want to say correct me from
long 60 to 90 seconds somewhere before
we would have damaged the interchanges
so the 45 have about 10 or changes
another 45 and we're ticking along it's
20 seconds 30 seconds and I should have
been going hey Chuck I should have been
more proactive I made the assumption
that he's looking the data he's singing
it - he's congressman put the fear of
God what do you realize he'd made a call
like what we're gonna do next and I'm
realizing he's not looking at the data
and I'm realizing I should have been
pushing it well I certainly should have
been paying attention to get the primary
action to ensure that it was done yeah
and so that was one of the things that I
wish I had done better back at the time
was to be more cognizant to making sure
that they took the action that we asked
I'm just taking too much it was my job
to watch the systems I should keep them
Kevin it's a team effort both ways
absolutely team effort all the way all
the way around good and bad we work as a
team
let's turn both
sound like he's getting little stressed
out there yeah I think the seriousness
of it was probably starting to soak in
yeah you probably I'm just guessing you
just realized I made a call and I'm
looking at the procedures and my call
didn't go go up and number less and I
need to push this thing go harder yes
yeah
something like that yeah still I think
the lesson to be learned there was give
a more exact call yeah and you're more
likely to get a quick response so for
you to call it because like you said
there was no panel there it's just the
pumps and if it was an ASP they might be
looking he's looking yeah okay the free
loops are off the cold slug is stopped
but it's still sitting there
a silent menace ready to wreak havoc the
team knows this but how does the team
keep the interchange from cracking and
still reestablish cooling now how much
time do they have so you got both frame
loops off and you know that's a drastic
action how long could the frame looks be
off and that you know and how do you
gauge sex it's a big deal to have in the
frame lips off did you are you starting
another timer because you were starting
your your interchange your fries timer
it's like I know I got so much time
before that cold slug hits now turn the
feelings off you got another timer going
now I didn't have a ton I didn't have a
timer guy I had the the the general
knowledge that things don't melt down
instantly right to me probably the fuel
cells or of the biggest concern and I
know that you know for fuel cells what
we could go for loss of coolant pumps
don't feel so we could go like nine
minutes or thereabouts so yeah I think
you have probably at least that order of
magnitude before something you know
catastrophic
so clearly I think you have not a lot of
time but an order of magnitude more time
to deal with them being off than to deal
with the cold slug coming around right
okay that kind of what did you have a
clock going in your mind at all well the
good thing is we had the EPS guys and
the fuel cells that I knew was the
shortfall we had the EPS guy washing the
fuel cells across there and he would be
the one to make the recommendation
whether we had to terminate the merge do
it and basically a power down of the
vehicle we couldn't gain it so now let
me ask you this have you guys seemed to
this case basically prior to this no I
don't remember no not really and the
other thing even when they did start
simulating this doing it a
simms of this later after this event for
future teams it one of the things about
the post line was always very stressful
for me is you can't recreate in training
the all the noises the ground convoy is
the confusion of the additional loops
the communication that's going all loops
your lives and listening to it was very
noisy it was always a very stressful
time for me when I became an e comment
what asset entry was because we didn't
train it and they were just not only
that even even afterwards was this the
chaos they everyone who's getting done
with her job in the control center it
was noisy in the control center it was
noisy on the loops and he had this
unknown factor what we don't train with
which is the KSC guys very often that we
we did start using direct comm with the
guys trying to minimize any kind of
problems we can directly talk to the
guys at the convoy but no it was the
training was we sat now I walked through
this procedure there was also a lot of
lessons learned like I said from sts-3
with things that they had done after we
handed over they did some additional
ammonia boiler tests that they under Tim
and there were some lesson learned from
that and the fact that that happened
there made it very cognizant
what happened once it could happen again
yeah I think I just can't emphasize
enough the few minutes of time that we
took preparing and walking through that
was just enormous hell I think it's as
much better is as I could have done in
that situation and I and I certainly
wish that I had I think I could have
done a lot worse had we not done that
preparation because turning off of freon
loops is a you want to do what yeah the
fact that we had talked through it it at
least it resonated with me that yeah
that's what we have to do we talked
through that we talked about the the
urgency of time and how that has to be
done to preserve that heat exchangers so
that I at least never questioned that
aspect of it had we not talked through
that case is is deeper respect as I have
for Quinn and knew that he was I had the
best thermal guy in in the back room I
don't think I would have just said okay
whatever you say I think I would have at
least questioned some and that might
have made the difference in the case so
so that preparation we we did was just
worth invaluable
now was that SOP for you like if you had
an event coming up that you would talk
to the backroom say this is the things
that could go wrong this is what we're
gonna do was that just SOP or what you
just kind of got lucky that time that
you did that you know I don't know it's
probably SOP when you feel like you
don't know as much as you should know
about something and and again coming
from the electrical side of the house
and trying to pick up the thermal and
life-support stuff you you don't pick up
all the knowledge you need in a matter
of a couple of years which is probably
as long as I had been thinking about it
so it's many years of a preparation to
really understand those systems so to me
it was basically all that we could all
that I could do personally in limited
time in order to be the most prepared
that I could so had I had five or ten
years experience and then I might have
been more complacent to not felt the
need to do that but I did it out of a
personal I have to cover myself right
right gotcha excellent let's keep going
here so we got that right now the free
loops are that's something
sounds like we you talked about this
previously so had you already stepped to
kind of through the recovery of yes
actually over time for him we actually
went through all the continued CG
aspects or recovery steps and kind of
walking through it was really uh so we
kind of do the plan that we're going to
do like again I was at the point
remember where those steps in the Indian
trajecsys Tyvek kilo to reactivate like
that yes okay that's good and so your
sounds like you're doing one loop at a
time why don't you just turn them you
know the under temps we've stopped it
let's go and turn them both on why did
you do it that way instead of one loop
at a time oh it was actually part of a
procedure but the real reason is is
you're owing your you're only activating
one so half the ones saying you're only
doing with half the coleslaw at a time
and the vehicles being hot right now
water loops we've got interchanges been
hot since it's been running without any
cooling so we trying to introduce one
loop look it over only add you know and
one of the cold slugs at a time and then
evaluate you know how the system is
working so you get the insight into the
situation whether or not you're under
temp situation is right basically
remedied right or not with with you
could look at as half the risk you know
half the heat rejection right quality
the heat exchanger and I wasn't paying
attention to gel pick where you're
picking the freon loop that was
associated with the ammonia system I
wasn't paying attention to that actually
Baltimore loops were off at the time so
I mean we turn it back oh you knew both
ammonia systems are off it didn't matter
right okay that's right okay do we have
an additional measurement on one yeah
you had an interchange are in on loop
one okay yeah so out of the fit swipe we
might have picked you might have picked
that one first
okay I just did what he says chuckles
good okay now I can tremor now if he
already gone flow proportioning valve to
payload at this point I can't actually
believe it or not the old procedure was
go to interchange er the assumption was
the payload loop was gonna be the so
slower flowing loops so with the
procedures written generically always go
to interchange
oh it's been changed and so right right
right so if you had a payload you would
freeze that payloads water if it was
there okay
did I catch that right y'all didn't turn
the water loop off unless they must have
said it wrong you may have said on well
maybe that's on yeah one of the points
to is in when you listen to this loop
it's amazing to me after listening to
this thing how I knew exactly what I'm
going to do but I was under the stress
in panic but under the stress I was
having trying to communicating exactly I
was having trouble between between 1 and
2 and what pump I was filling to
communicate Chuck did an excellent job
it's ok
I heard what you said is this really
what you wanted were you saying and it
was very easy to go back and say yes
that's what I meant to say but it was
amazing how simple things like loop 1
loop 2 even they had the procedure over
I was saying the wrong thing he just
gonna stress it and I think Chuck said
you had you got yourself out of the
quick response situation by turn of the
friend loose off now you had kind of an
order of magnitude more so you could
slow down a little bit right a little
you know yeah not a lot you got the fuel
cells out there heating up but yeah you
could able to slow down a little so does
that go through your mind ok I can take
a little bit more time make sure I'm
understanding the you know pedal to the
metal in your mind I think it was
okay I never heard that before until you
mentioned
at that point you had one friend loop up
and I heard the EPSS mentor right when
that Larry men are back there saying the
fuel cell returned temps starting to
skyrocket at that point with one
friendly pup were y'all feeling pretty
good about yourselves or did you need to
keep going get both loops up to you know
get yourself in a really good config or
what was your clock like or starting to
get over the hump so I know you just
said it there but just to put it out
there so at that point what was the
problem and how did this whole thing
kind of evolve they did a new when they
did a new procedure at on the cooling
cart with a research fluid
I believe research recirculated the
fluid in the cooling cart it would allow
this old slug to build up and so when
they initiated they got this really cold
slug to occur and I think that was it
was a but there was a new procedure they
implemented it for 27 and I think they
didn't change it back after 27
although standard procedure they had was
it back then was it standard to have the
ammonia in the ground cooling but at the
same time they both came up at the same
time right he had ammonia and they
hooked up is that was that standard back
then normally they would notify us ahead
of time that they're ready to initiate
at that point we would make the call at
deactivate the ammonia and then they
would initiate in this case they
initiated
I don't know wasn't because it's a new
procedure researched and this Cole
suddenly initiated it up but we were not
notified that they were about to
initiate it and that there was also
there was some pressure that they knew
that we're getting near the end of our
second ammonia tank right and there has
there being a schedule right there to
get scheduled pressure to get get that
initiated so I think that they got in a
hurry and they initiated without going
to mal seizure with a contact and say
I'm more about to initiate wow so
there's another another piece of an
error change you got some some guys on
the other end her feelings and schedule
pressure we've got some procedures that
weren't synced up to calm but wasn't
synced up people starting to get
distracted with the plaque King there
was a lot of a lot of variables going on
in there and I captured that really well
I didn't realize that that all that was
going on and even listening to the to
the recording I didn't realize you could
guys had you guys had obstacles out in
front of you didn't know about right
yeah that's really good in hindsight is
always 20/20 I I wish that I would have
done
KSC end to have a better handoff and I
think we did improve on that a lot after
after that right right let's see what
else so sounds like EPS is freaking out
a little bit but you know it's it's
gonna get better here I was like I I
don't know if it was out of ignorance or
not but I was less concerned about it
then then Larry was at the at the time
yes and you knew you had you had one
friendly pup senior you had some cool
you remember what the kW was about that
time I think he was saying for about
four probably are never fuel stuff for
that fuel cell right I see but so if
there even is probably about 12 or so
right yeah so one friend lives should be
able to handle handle 12 pretty well
okay
yeah
here we see cooling on now or go to
think free only to to pump I just I
couldn't take freon pump so as quickly
as it came and almost subsided and
forget all about you right
yeah like when said that since that was
the Vox I think the pain lasted a little
bit yeah this is a box this is what we
got about nine and a half minutes there
so unboxed how long do you think the
situation really went about double that
maybe your you'll recall that's probably
yes right let's excellent any other
summary thoughts about the situation he
gave a lot of hindsight while you were
well we were going but any any other
thoughts about the situation I guess
some of the things that came from this
mission and like we talked about like
rule one last one got changed we started
to Train post-landing we spent a lot of
time with the KSC folks trying to
establish better protocols and
communication between the e common
ground and the ground convoy guys we we
improved the about t-mobile procedure a
lot of improvements on how to go through
it what else to do we your Miami stuff I
long there was a lot of improvements on
how oh and the communication with the
flight director stressing with them that
hey look at the vehicle here's what
under temp occurs this is why we're so
scared of it and they in the flight
Richards in a lot
better job post-landing to make sure the
team is cognizant hey not me ain't
handing over until keep the room quiet
until around who's established and the
crews out itself sometimes even when the
missions seemingly over it may not be
over in the midst of distraction the
EECOM team identified the under temp
situation diagnose the cause of two
cooling systems running simultaneously
then resolve the situation while
preserving crew safety and all vehicle
hardware the mcc flight controller is
one who must always be aware that
suddenly and unexpectedly we may find
ourselves in a role where our
performance has ultimate consequences
discipline competence confidence
responsibility toughness teamwork and
vigilance these are our core values
values that have served us well
values that have allowed us to slip the
surly bonds of Earth and perform our
dance above the skies
you